Objective

To explore the characteristics the eLife data. It is structured into two major parts:

  1. Univariate Exploratory Data Analysis
  2. Bivariate Exploratory Data Analysis

Libraries

Globals

Univariate Exploratory Data Analysis

Text Data — Editor (Initial) Consultations

Basic Info

Initial consultations — when an editor asks another to vet a submission — come in 10 files. These files were messy (with comment fields erroneously spliced, etc.), so I had to clean and merge them (see merge_consultations notebook).

Explorations

→ There about 257K comments during the editor consultation.

→ For about 46K initial submissions, editors consult.

→ A typical editor consultation comment is 70 tokens.

These are shorter than reviewer consultation (below). These don't include the editorial letter draft, which editors often copy and paste into the reviewer consultation exchange.

Zoomed into to a smaller range of n comments:

→ Typically 3 editors are involved.

→ There are 1318 manuscripts with just 1 consulting editor.

... Which would imply no consultation. So these might need to be explored further.

Text Data — Reviewer Consultations

Basic Info

The reviewer consultation data comes in 10 files, 2012–2022, post-review. See merge_consultations notebook for how I cleaned and merged these files. There are ~144 K consultation threads with 5 features.

Explorations

→ There are about 145K comments during reviewer consultation.

→ Reviewers usually always consult after they reviewed a mansucript.

This number accords with the number of manuscripts sent out for review.

→ A typical reviewer consultation comment is 120 tokens.

→ A typical reviewer consultation has 7 or 8 comments.

→ Typically 2 or 3 reviewers are involved in a consultation.

... This is one less than the mean, because the editors are counted in these numbers, too.

Text Data — Reviews

Basic Info

The eLife review data contains only information about the reviews themselves. There are ~50K reviews (with manuscripts often recieving more than 1 round of ≥1 review) and 7 variables.

Here're the columns in the data. The ratings are predicted using roBERTa. See "classify_eLife" notebook for those procedures.

Explorations

→ Most manuscripts (MS) get 2-3 reviews.

→ Most reviewers in the eLife data review 1-3 manuscripts. There are outliers, though, probably editors.

→ Most reviews are around 500 tokens.

Zoomed in at the beginning of the range:

→The predicted ratings follow a normal distribution.

I pretrained roBERTa on ~28K ICLR reviews. Their ratings were 0-5, in the same ordinal direction (=5 outstanding)as the labels I used for elife. Then, I pretrained on my eLife labels, which were 1-4. Finally, I predicted all of eLife's reviews. The final the model produced labels for eLife that were just below and just above my hand label range. I thought about rescaling ICLR but I didn't want to lose any nuanced meaning in that data. But I didn't think the difference in the two scales' magnitudes would be all that important; the theory was that the relationships between the texts and the rating scales in either was consistent and generalizable to both.

Pipeline Data — Manuscripts' Paths

Basic info

The manuscript data contains all manuscripts, reviewed and desk rejected. There are about 61 K manuscripts with 25 features.

Here are the variables in the manuscript data. *_dt corresponds to "date".

Explorations

Missingness

Random fake rows in the dataframe.

MS Pipeline

→ About 60K manuscripts were submitted to eLife

Reviewed manuscripts

→ About 19K MS actually get reviewed.

Dichotomize Final Decision

→ Of those MS reviewed, more than half eventually get accepted.

Demographic Data — Reviewers

Basic info

The reviewer data contains all demographic info on the reviewers. There are about 53K reviewers and 8 columns. If elife editors are acting as reviewers (and not managing the review of an MS), they are indistinguishable as editors from reviewers using Reviewer ID. They can be differentiated by name and institution, though.

Here are the features we have on reviewers.

Infer reviewer location

The reviewer data doesn't come with reviewers' countries. To infer where they're at, we take their email TLDs and reverse-look up the country.

Import country using a {country -> TLD} mapping downladed from github.

Grab domain, infer country, determine if in commonwealth

Demographic Data — Authors

Basic info

Author data contains all demographic information on all submitting authors, including co-authors. In the analyses below, I summarize all authors and then just the "corresponding authors" — these are not necessarily the first author on the paper.

Here are the features we have on authors. Note that country is input by authors themselves and varies dramatically (i.e. US, USA, the US, United States). This gets a bit tricky down the line.

Infer author location

Infer Auth/Rev Diversity

Here I look at country of origin (i.e. where authors write from) and conventional gender identity of the name authors write under.

Generate measure of country diversity

I compute the country_diversity as the ratio of authors from unique countries over the total number of authors. So, if there are 4 authors and 3 come from different countries, diversity = 3/4. In single-author papers, the ratio would be 1/1, so I hard code these as 0 (i.e. no diversity).

Infer world region from country

So we don't inefficiently look up a country repeatedly (which is slow with requests, I create dict containing 1 country as the key and its world region as the values. Then I use this dictionary to populate the region field on the original dataframe.

Here we construct indicators of the world region and whether contributor is based in Americas/Asia.

Infer gender from first name

With the first names, we use genderize.io to classify the names' conventional genders.

Un-comment to re-do gender classification (takes over an hour):

Here, I compute a gender spectrum based on the first name.

Explore Contributor Diversity

Authors

→ Over half of all submitting authors are in/from the US and UK.

→ 15% of all submitting authors are in/from Asia.

→ Over 56% of all submitting authors are in/from the Commonwealth.

→ Over 61% of all submitting authors are men.

→ 53% of corresponding ("first") authors are from US and UK.

→ 59% of corresp. authors are from/in the Commonwealth

→ 14% of corresp. authors are from/in Asia

→ 69% of corresp. authors are men

Reviewers

The countries of reviewers who have email TLDs .com or .org are classified as "unknown" since these are not linked up to a specific country.

→ Over 55% of all reviewers are in/from the US & UK.

→ 6% of all reviewers are in/from Asia.

→ 61% of all reviewers are in/from the Commonwealth.

→ 73% of all reviewers are men.

Co-Author Collaborations

These stats jointly describe the authors collaborating on a given manuscript.

→ Most manuscripts have no collaborating authors from Asia.

→ By and large, collaborators tend to either be all from Commonwealth or all from other places. The majority of manuscripts don't have mixed teams.

→ Most manuscripts are mono-national, i.e. authors tend to be based in the same country.

→ Many papers have no women co-authors. But the distribution sits at around 33%, i.e. 1 woman collaborator to every 2 men.

Bivariate Exploratory Data Analysis

Editorial Outcomes by Author Characteristics

Initial Decisions

→ No big differences in initial decisions for authors based in Americas vs. other places.

→ No big differences in initial decisions for authors based in Asia vs. other places.

→ No big differences in initial decisions for women vs men authors.

→ No big differences in initial decisions across regions.

Africa/Antartica have v. few authors.

Final Decisions

→ No big differences in final decisions for authors based in Americas vs. other places.

→ No big differences in initial decisions for authors based in Asia vs. other places.

→ No big differences in final decisions for women vs men authors.

→ No big differences in final decisions for authors across regions.

Reviewer Ratings by Author Characteristics

Location

→ Reviewers tend to be harsher on authors from Asia (but the SD is huge).

→ Reviewers tend to be harsher on corresp. authors from Asia (SD is huge)

→ Reviewers tend to be harsher on corresp. authors outside of Commonwealth (SD is huge)

→ No relationship between average country diversity of authors and average reviewer rating.

Note this is asking: Do diverser manuscripts get better or worse reviews, on average. No relationship.

Gender

→ Reviewers tend to rate women and men authors the same.

→ Reviewers tend to rate women and men corresp. authors the same.

→ No relationship between gender diversity (= prop. of women authors) and average reviewer rating.

Reviewer Ratings by Review & Reviewer Characteristics

Review Length

→ Longer reviews tend to be more negative.

Location

→ Reviewers based in Africa and Asia tend to rate less harshly; In Europe most harshly.

→ Reviewers based in the Commonwealth tend to rate the same as those elsewhere.

Gender

→ Women and men reviewers tend to rate the same.